The present invention relates generally to software system performance diagnosis, and more particularly, to computer assisted hot-tracing mechanism.
The following background references are discussed herein:    [1] gdb: The GNU Project Debugger.    [2] Nicholas Nethercote and Julian Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation (PLDI '07). ACM, New York, N.Y., USA, 89-100.    [3] Process Trace Linux (PTrace).    [4] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, PLDI '05, pages 190-200, New York, N.Y., USA, 2005. ACM.    [5] V. Prasad, W. Cohen, F. C. Eigler, M. Hunt, J. Keniston, and B. Chen. Locating system problems using dynamic instrumentation. In Proceedings of the 2005 Ottawa Linux Symposium (OLS) (July 2005).    [6] B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In Proceedings of the annual conference on USENIX Annual Technical Conference, ATEC '04, pages 2-2, Berkeley, Calif., USA, 2004. USENIX Association.    [7] Úlfar Erlingsson, Marcus Peinado, Simon Peter, and Mihai Budiu. 2011. Fay: extensible distributed tracing from kernels to clusters. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (SOSP '11). ACM, New York, N.Y., USA, 311-326.
Large-scale complex systems often have hidden bugs/mis-configurations etc., which are difficult to detect. This problem is further compounded by the fact that many applications are implemented at large scale, and cannot be tested in real world scenarios during the development stage. Hence, bugs or resource mismanagement problems are almost impossible to replicate in a debugging environment. As systems become more complex monitoring solutions are required to be enable a deeper analysis of the system.
While a lot of monitoring solutions are available in the market, most of these suffer from either a heavy overhead, or run on predefined functions hence reducing usability. Alternatively, most solutions also require changes to the kernel or application itself to allow for implementation.
Conventional debugging techniques by such as GDB [1], valgrind[2], ptrace[3], and pin[4], run the entire process in debug mode, allowing developers to oversee the entire execution of the process. Hence each instruction executed in the application can easily be captured backtracked and scrutinized. However, such an approach enforces a considerable performance overhead, which is not practical for deployed systems. It is also not possible to observe the true behavior of the application when executing in the debug mode as it considerably changes the behavior of the application. Hence a very lightweight and non-intrusive approach is needed.
Other modern approaches such as those followed by System Tap[5], only capture kernel level events, which are not often not useful from the perspective of a debugger. The D-Trace[6] architecture attempts to capture all level events but uses a h/w trap mechanism to context switch to kernel space, this results in a heavier overhead. A lesser known technique applied by Fay[7], leverages the fact that recent windows binaries have a free space at the start instructions which it uses to make a trampoline function based instrumentation. While trampoline mechanism also has an overhead (albeit significantly less than D-Trace), this approach is restricted to windows where the native binaries support such a mechanism. Another key limitation of these approaches is that they need a special infrastructure/API to be installed in the kernel, which acts as a middle layer interface, compared to our technique, which will allow applications to execute in the native environment.
The closest work to I-Probe is DTrace[6] and Fay[7], shown in FIG. 6 (prior art) the key differences/novelties of I-Probe compared to these approaches:                1. Zero-Effect and Zero-Overhead hot patching mechanism:                    A zero probe light-weight compiler assisted hot-patching mechanism. The key idea being to leverage existing compiler technology, to allow for very light-weight hot-patching instrumentation in large scale systems.                        2. Platform-indepent, and no changes required in the kernel:        
Since the inventive I-Probe is compiler driven it is platform independent, it simply needs exiting compiler flags/options to introduce calls to functions, which can be used as placeholders. This is available in several compilers such as visual C++, and GNU compilers. On the other hand similar infrastructures like D-Trace nor Fay are platform dependent as they need to introduce changes in the kernel as they either need a special b/w trap mechanism or a special binary format which already has empty space (windows).                3. Completely inline hot-tracing:        
The invention's instrumentation is completely inline, hence it avoids overhead similar to kernel trap mechanisms in D-Trace, or alternatively trampoline mechanisms in Fay, this makes it extremely efficient with lesser overhead and overall a simple and clean implementation.                4. User-space only logging:        
The invention's instrumentation has a user-space only logging, hence there is no unnecessary context-switch between the process and the kernel.